Tag: FP8 training

Kernel optimization for LLM training: techniques, trade-offs, and use cases

Which kernel-level optimization techniques most improve LLM training efficiency, and what are the trade-offs and use cases of each?

Kernel-level optimization is where most LLM training efficiency gains of the last six years come from, and the field splits into five families: fused attention kernels, low-precision GEMM kernels, mixture-of-experts kernels, communication-overlap kernels, and fused elementwise/optimizer kernels. The evidence is consistent that each family delivers large, workload-specific gains — 2x on attention, 25-75% on FP8 training, 1.4-5.7x on MoE layers, 1.2-2.9x on long-context training — but the gains are conditional on hardware generation, model architecture, and engineering effort, and almost every headline number is vendor- or single-group-benchmarked. The main caveat: no shared benchmark exists, so cross-paper 'x-times faster' claims are not directly comparable.

GPU kernels · LLM training · FlashAttention · FP8 training · Mixture-of-Experts · kernel fusion · communication overlap